System and method for the creation, synchronization and delivery of alternate content

ABSTRACT

According to one embodiment, A nonlinear editing system comprises a video encoder, an audio compressor and an engine unit. The engine unit is adapted to synchronize the video with the first alternate video and to synchronize the audio with the first alternate audio so that, where the scene of the program includes objectionable content, either (i) the video is substituted with the first alternate video, (ii) the audio is substituted with the first alternate audio, or (iii) both the video and the audio are substituted with the first alternate video and the first alternate audio.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority on U.S. ProvisionalPatent Application No. 60/636,504 filed Dec. 15, 2004 and U.S.Provisional Patent Application No. 60/637,305 filed Dec. 16, 2004, thecontents of which are incorporated by reference.

BACKGROUND

1. Field

Embodiments of the invention relate to the field of digital contenttransmissions and display. More specifically, one embodiment of theinvention relates to a system and method for creation, synchronizationand delivery of alternate content for dynamic substitution of unwantedvideo or audio on a scene-by-scene basis.

2. General Background

Television is used to deliver content, such as entertainment andeducational programs, to viewers. A growing number of parents are nowwatching and listening to content perceived by their children, in aneffort to better shield their young children from exposure toobjectionable content, namely violence, sexual situations, indecent orsuggestive language and the like.

In television programming, for example, the removal/restoration ofpotentially objectionable content is implemented during contentauthoring/editing using a nonlinear editing system.

As illustrated in FIG. 1, a conventional nonlinear editing system 100 isused to select desired portions of audio and video sequences (scenes)taken from one or more libraries 110, which contain all of the rawcontent footage (e.g., video, audio, etc.) under consideration, and thento establish their temporal relationships. The temporal relationshipsinvolve the arrangement of adjacent sequences of the same content type.

For instance, as shown in FIG. 2, video sequences 202, 204, 206 and 208,namely uncompressed digital video 200, are arranged in a temporalrelationship to each other. Audio sequences 212, 214, 216 and 218,namely audio 210, are similarly arranged with respect to each other.Thereafter, each video sequence 202, 204, 206 and 208 is synchronizedwith one or more audio sequences which may represent different forms ofaudio such as dialog tracks 212 and 214, music track 216, sound effects218, or the like. Normally, the audio is in the form of Pulse CodeModulation (PCM) audio samples. Even though the end product consists ofone continuous video sequence with a single synchronized audio track(containing a composition of multiple forms of audio accomplished by adigital mixdown unit 120 of FIG. 1), all digital components 200 and 210comprising the content remain distinct and separate while in nonlinearediting system 100.

Referring back to FIG. 1, after the final edited version of the contentis completed, it is then assembled into a compilation including theselected video and audio sequences. This involves the selected video andaudio sequences being compressed, perhaps using MPEG-2 video encoder 130and AC-3 audio compressor 140 for video and audio respectively, andsynchronized with each other to produce a continuous transport stream150 of a reduced size. This size reduction is what makes thetransmission, broadcast and/or storage of digital video contentpractical, allowing it to fit on inexpensive media like DVDs or to becarried in standard six megahertz (6 MHz) Radio Frequency (RF) spectralchannel concurrent with eight or more other A/V services.

After compression and placement into transport stream 150, the video andaudio cannot be edited. Transport stream 150 features video running at aconstant frame per second rate and compressed audio synchronized withthe video. The audio is continuous as well, where silence is representedby specific data values. Conventional nonlinear editing system 100offers no mechanism to produce any transport streams with alternatecontent that can be substituted for selected video and audio.

At the destination, with respect to the rendering of video programs,televisions have been implemented with technologies that allow parentsto preclude the playback of programs that exceed a selected age-basedrating. A black screen is shown for any program that exceeds theselected age-based rating in order to prevent such viewing. This programblocking can be disabled with a parental password.

However, there are a number of major shortcomings of these technologies.For instance, the filtering only occurs at the program level, namely aviewer gets to view the entire program or nothing at all. In otherwords, there is no selective blocking capability, and clearly no abilityto substitute alternate content provided with the transport stream.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are illustrated by way of example and notby way of limitation in the figures of the accompanying drawings, inwhich like references indicate similar elements and in which:

FIG. 1 is a block diagram of the operations of a conventional nonlinearediting system.

FIG. 2 is a block diagram of a display of content processed by theconventional nonlinear editing system.

FIG. 3 is a block diagram of a display of content processed by a dynamicnonlinear editing system.

FIG. 4 is a block diagram of the operations of the dynamic nonlinearediting system developed in accordance with one embodiment of theinvention.

FIG. 5 is an exemplary block diagram of a content delivery networkconsistent with certain embodiments of the invention.

FIG. 6 is an exemplary block diagram of a content delivery back-endconsistent with certain embodiments of the invention.

FIG. 7 is an exemplary diagram of a content filter unit implementedwithin a digital device of the content delivery system of FIG. 5.

FIGS. 8A and 8B are exemplary diagrams of an Adaptation field of anI-frame, being a portion of a MPEG packet of the digital data stream.

FIG. 9 is an exemplary diagram of a first embodiment of a contentreplacement process consistent with certain embodiments of theinvention.

FIG. 10 is an exemplary diagram of a second embodiment of a contentreplacement process consistent with certain embodiments of theinvention.

FIG. 11 is an illustrative flowchart of a process for determiningwhether a segment of content associated with a program should beprevented from being displayed.

FIG. 12 is an illustrative flowchart of a process for substitutingreplacement segments of content based on parental control parameters.

FIG. 13 is an exemplary embodiment of a Age-based Rating scheme formodification or substitution of scenes including objectionable content.

FIG. 14 is an exemplary embodiment of a Content-based Rating scheme formodification or substitution of scenes including objectionable content.

FIGS. 15A and 15B are exemplary embodiments of a Tag Rating scheme formodification or substitution of scenes including objectionable content.

DETAILED DESCRIPTION

Various embodiments of the invention relate to a system and method forcreation, synchronization and delivery of alternate content for dynamicsubstitution of unwanted video or audio on a scene-by-scene basis.According to one embodiment of the invention, a nonlinear editing systemsupporting dynamic content replacement (hereinafter referred to as“dynamic nonlinear editing system”) is implemented within the contentdelivery network. At a digital device, a content filter unit isimplemented within the digital device and is configured to provideparental control by performing scene-by-scene evaluation of contentassociated with the incoming transport stream (e.g., video, images,audio or any combination thereof) and subsequent modification orelimination of objectionable content. Of course, the content filter unitmay be implemented at the front-end in lieu of at the back-end that isdescribed below in detail.

In short, the combination of the dynamic nonlinear editing system andthe content filter unit provides scene-by-scene filtration, and even tothe granularity of frame-by-frame, to block or replace individual scenesor words. Such blocking may be accomplished by the content filter unitresiding with the customer through screen blocking or obscuring pixelsfor a particular image or muting audio. Alternatively, content withinthe scene can be replaced in order to reduce the rating of the program(referred to as “down-rating”), or may even be replaced to reinsertcontent that was previously obscured/removed in order for theprogramming to achieve the higher rating (referred to as “up-rating”).

According to one embodiment of the invention, the replacement content istransmitted in the same digital transport stream as the primary content,and is synchronized with the primary content that it is replacing forswapping on-the-fly. Thus, the rendering of higher or lower ratedcontent is accomplished, where programs can be created and broadcastwith multiple versions having different content and ratings. Also, allcontent, including advertisements, news, sporting events and the likemay be filtered as well in lieu of just movies and television shows.

For purposes of the present description, the term “digital device” mayrefer to consumer electronics that is adapted to tune, receive, decrypt,descramble and/or decode transmissions from any content provider. The“consumer electronics” may include, but is not limited or restricted toa television, a set-top box, a personal digital assistant (PDA), acomputer, a cellular telephone, a video game console, a portable musicplayer, a stereo system, a personal video recorder, or the like.Examples of “content providers” may include a terrestrial broadcaster, acable or satellite television distribution system, or a companyproviding content for download over the Internet or other InternetProtocol (IP) based networks like an Internet service provider.

In the following description, certain terminology is used to describefeatures of the invention. For example, in certain situations, the terms“component,” “unit” and “logic” are representative of hardware and/orsoftware configured to perform one or more functions. For instance,examples of “hardware” include, but are not limited or restricted to anintegrated circuit such as a processor (e.g., a digital signalprocessor, microprocessor, application specific integrated circuit, amicro-controller, etc.). Of course, the hardware may be alternativelyimplemented as a finite state machine or even combinatorial logic.

An example of “software” includes executable code in the form of anapplication, an applet, a routine or even a series of instructions. Thesoftware may be stored in any type of machine readable medium such as aprogrammable electronic circuit, a semiconductor memory device such asvolatile memory (e.g., random access memory, etc.) and/or non-volatilememory (e.g., any type of read-only memory “ROM”, flash memory, etc.), afloppy diskette, an optical disk (e.g., compact disk or digital videodisc “DVD”), a hard drive disk, a tape, or the like.

In addition, the term “program” generally represents a stream of digitalcontent that is configured for transmission to one or more digitaldevices for viewing and/or listening. For instance, the program may beconfigured as a transport stream. According to one embodiment, theprogram may contain multiple identifiers such as Packet Identifiers(PIDs) when the program is MPEG (Moving Pictures Expert Group) compliantcompressed video. These multiple PIDs provide for replacement of onescene of content for another. The “scene” is defined as one or moreframes of content, namely video, images, audio or any combinationthereof. For instance, a scene may be one or more frames of video and/oraudio, or even a packet of music downloaded into a portable musicplayer.

More specifically, multiple PIDs may be associated with a single programin order to provide content replacement for this program. Contentreplacement provides an enhanced level of viewing/listeningcustomization. For example, content replacement can be used to providetargeted advertising to an audience by the substitution of oneadvertisement determined to be “objectionable” for another. Contentreplacement can also be used to provide multiple scenes, endings orother segments for a program, or perhaps to provide multiple views in asportscast or other program. Other applications for the contentreplacement of the invention fall within the spirit and scope of theinvention.

Certain types of content, such as movies, have an associated contentrating established by a supervisory body to inform potential viewers ofthe type of material that may form a part of the content. Generalratings of G, PG, PG-13, R and NC-17 are commonly established for mostmovies produced in the United States by the Classification and RatingsAdministration (CARA) of the Motion Picture Association (MPA) or theMotion Picture Association of America (MPAA). Television programming issimilarly rated by the TV Parental Guidelines Monitoring Board, whichprovides ratings of TV-Y, TV-Y7, TV-Y7-FV, TV-G, TV-14 and TV-MA.

For purposes of this document, content having a rating provided undersuch rating systems will be referred to as having a “higher” rating ifit is intended for more mature audiences, and a “lower” rating if it isintended for a more general or less mature audience. Hence, thepreviously described rating systems are listed from lowest to highest.Therefore, within this terminology, an R rating would be considered a“higher” rating than a G rating. Such nomenclature is defined herein forsimplicity of description of the above rating system as well asequivalent systems, without intent for the nomenclature to pass anyjudgment on the quality of such content.

Additionally, the term “rating” is intended to embrace any suitablecontent rating system, domestic or foreign, public, private or evengovernmental that is formed to serve a similar function. Such ratinginformation may be provided as data embedded within program specificinformation such as data inserted into an Adaptation field of a MPEGpacket for example when the content is video.

While this invention is susceptible of embodiment in many differentforms, there is shown in the drawings and will herein be described indetail specific embodiments, with the understanding that the presentdisclosure is to be considered as an example of the principles of theinvention and not intended to limit the invention to the specificembodiments shown and described.

I. General Architecture of Content Delivery Front-End

In order to support dynamic content substitution, the authoring processassociated with the dynamic nonlinear editing system is modified toallow concurrent editing of one or more alternate video tracks andadditional audio tracks. Each of the alternate video tracks issubordinate to the primary video track and is temporally synchronizedwith it. After post-processing, these video tracks will be carried inthe same transport stream, namely as part of the same program. Thedynamic nonlinear editing system is extended to accommodate theadditional tracks for alternate video and audio and is complementary tothe editing paradigm established for this tool.

As described earlier, the “final cut” is assembled, mixed (audio) andstreamed to compression equipment (encoders). A video encoder can onlyaccept a single, continuous video stream. The primary video meets thecriterion. The alternate video, however, is normally a non-continuoussequence of video that is transmitted concurrently with the primaryvideo and, in some situations, used to replace the primary video orportions thereof.

Referring to FIG. 3, a block diagram of a display of content processedby a dynamic nonlinear editing system being part of a content deliveryfront-end is shown. The dynamic nonlinear editing system 300 is modifiedto support synchronization and delivery of replacement content, such asalternate video 315 and/or audio tracks 360 for example.

As shown, alternate video 320 is provided during a first time period 312(from T2 and T3) of a master timeline 310, which corresponds to thedisplay duration of video scene 330. Also, alternate video 340 isprovided during a second time period 314 (from T6 and T7) of mastertimeline 310, corresponding to the duration of video scene 350. Theduration of first and second time periods 312 and 314 may constitute thedisplay time for one or more frames of video (e.g. 15 frames of videoconstituting one-half second of display time, or perhaps more frames).

As further shown, alternate audio 360 includes multiple audio segments362-366. A first alternate audio segment 362 occurs at the start of timeT0 and ends at time T0 _(A) while a second alternate audio segment 363occurs at time T2 _(A) and ends at time T3. Between time periods T4 andT5, a third and fourth alternate audio segments 364-365 are placed,namely between T4 _(A)-T4 _(B) and T4 _(C)-T4 _(D). Similarly, a fifthalternate audio segment 366 occurs at time T6 _(A) and ends at time T6_(B). For this embodiment, first, fourth and fifth alternate audiosegments 362, 365 and 366 constitute alternate dialog. Second audiosegment 363 constitutes an alternate music track and third audio segment364 constitutes an alternate special effects track.

Referring to FIG. 4, a block diagram of the operations of dynamicnonlinear editing system 300 developed in accordance with one embodimentof the invention is shown. Dynamic nonlinear editing system 300 receivesinformation from content storage units 410 and 412, which are configuredto store video and audio content. Dynamic nonlinear editing system 300is adapted to sequence video and audio retrieved from content storageunits 410 and 412, and then to establish their temporal relationships.The temporal relationships involve the arrangement of adjacent sequencesof the same content type.

In order to remain compatible with video encoder 420, dynamic nonlinearediting system 300 fills the time periods before, after and betweenalternate video 320 and 340 with synchronized black screen in order tocreate a continuous video stream for processing by video encoder 420. Itis contemplated that the encoding of the primary and alternate videowill occur on the same encoder, and therefore, the two processes willoccur serially. As a result, dynamic nonlinear editing system 300 willneed to communicate with video encoder 420 so that synchronizationinformation may be inserted by video encoder 420 into both resultantcompressed video streams 430 and 440 for post-encoding reprocessingwhere video streams 430 and 440 are combined with propersynchronization.

Similarly, dynamic nonlinear editing system 300 fills the time periodsbefore, after and between alternate audio segments 362-366 with mutedaudio in order to create a continuous audio stream, for processing byaudio compressor 450. It is contemplated that the encoding of theprimary and alternate audio may occur serially on the same audiocompressor 450, or on multiple audio compressors. When using the sameaudio compressor 450 as shown, dynamic nonlinear editing system 300 willneed to communicate with audio compressor 450 so that synchronizationinformation may be inserted by audio compressor 450 into both resultantcompressed audio streams 460 and 470 for post-encoding reprocessing.

A transport processing engine unit 480 receives video streams 430 and440 and audio streams 460 and 470. Transport processing engine unit 480conducts a trimming operation of alternate content as well as performscontent synchronization, content re-multiplexing and meta datainsertion. These operations are complementary with the functionality ofdevice 510 of FIG. 5, and perhaps substantially similar where contentreplacement is conducted in the front end.

First, the alternate content contains blank video (black screen) andmuted audio between alternate content segments. This is a byproduct ofpreparing the content for compression. Transport processing engine unit480 trims all black screen content and muted audio purposefully added tocreate the continuous audio. This allows the alternate content to bemultiplexed into the primary transport stream.

As an illustrative example, referring to FIG. 3, video scene 330 andalternate video 320 need to be processed to ensure that these videosegments are temporally equivalent within the transport stream. In otherwords, these video segments need to be virtually identical in durationand aligned temporally where such editing occurs after beingencoded/compressed and during placement within the transport stream. Thesame temporal alignment is required for video scene 350 and itsalternate video 340.

Secondly, transport processing engine unit 480 uses synchronizationmarks inserted by video encoder 420 and audio compressor 450 to ensurethat the digital device will display either the primary or alternatevideo at the same time. As an example, for MPEG transmissions, threesynchronization elements are used for video: PCR (primary clockreference)—clocking at approximately 27 megahertz (MHz); DTS (decodingtime stamp)—placed on video itself to start conversion from MPEG toanalog video format; and PTS (presentation time stamp)—when pictureshould be displayed. Both primary video 330 and alternate video 320should have the same synchronization marks.

Similar synchronization marks may be used for audio, and thus, theprimary and alternative audio will have the same synchronization marks.This allows alternate content to be correctly located temporally withinthe transport stream so that primary/alternate video as well as theprimary/secondary audio can be contextually located adjacent to eachother.

Third, transport processing engine unit 480 merges the alternate contentinto the transport stream and inserts meta data as needed to enablesuitably equipped playback devices, such as digital device as describedbelow, to dynamically select any combination or primary/alternatecontent during broadcast or playback of the resultant composite content.As part of the merging process, the re-multiplexer corrects PCRsynchronization marks and other tasks normally encountered andassociated with the re-multiplexing process.

According to one embodiment, the meta data is inserted into anAdaptation field of one or more MPEG-based packets forming the transportstream. The meta data may provide Age-based rating information, contenttag information or the like as described below.

The composite, homogeneous output 490 from transport processing engineunit 480 is then returned to the normal content process flow, where itis stored or forwarded to a distribution phase by a content provider asdescribe below.

II. General Architecture of Content Delivery Back-End

Referring to FIG. 5, an exemplary block diagram of a content deliveryback-end 500 consistent with certain embodiments of the invention isshown. Content delivery back-end 500 comprises a digital device 510 thatreceives digital content such as a program from one or more contentproviders 520. The program may be propagated as a digital data streamfor example in compliance with any data compression scheme. Examples ofa data compression scheme include, but are not limited or restrictedMPEG standards.

According to one embodiment of the invention, digital device 510 decodesan incoming program, and evaluates the content of the incoming programon a scene-by-scene basis to determine whether such scenes include anyobjectionable content. Such evaluation may involve a review of therating assigned to each scene or perhaps access data within tagsassigned to each scene to determine whether the scene includes anyobjectionable content. These evaluation schemes are described below.

Since digital device 510 may be adapted to increase or decrease therating of the program, the term “objectionable content” has twomeanings. One meaning is that it is content featuring certain subjectmatter that may be inappropriate for the viewing or listening audienceat this time. Another meaning is that it is low-rated content that canbe substituted for higher rated content due to the mature nature of theviewing or listening audience and their request for up-rating ofcontent.

Content provider 520 provides the digital content to digital device 510through transmission medium 530, which operates as a communicationpathway for the program within the content delivery network. Thetransmission medium 530 may include, but is not limited to electricalwires, optical fiber, cable, a wireless link established by wirelesssignaling circuitry, or the like.

Content provider 520 may be adapted with a satellite antenna 600 toreceive a multiplexed stream of content from a satellite transmission asshown in FIG. 6. The stream of content is received, demodulated anddecrypted at a satellite receiver 605 before being routed to PID Mappinglogic 610. The stream of content may include replacement contentassociated with secondary PIDs. Alternatively, the replacement contentmay be retrieved from a local content database 615 or other sources ofcontent.

PID mapping logic 610 maps the incoming content from whatever source toa set of primary PIDs and a set of secondary PIDs. The incoming contentincludes (i) content originally intended for rendering (referred to asthe “primary content”) and (ii) substitution content for various scenes(referred to as “replacement content”). The primary content isassociated with the primary PIDs and the replacement content isassociated with the secondary PIDs. According to this embodiment, PIDmapping logic 610 may also be adapted to insert flags (or markers) intothe content in order to identify the location where replacement contentis to be inserted.

The mapped content then is routed to PSI/PMT insertion logic 620 thatinserts Program Specific Information (PSI) and Program Map Tables (PMT)into the content for use by the decoding side in decoding theprogramming. If the content is to be encrypted, it may optionally passthrough encryption logic 625 prior to modulation at a modulator (e.g., aQAM modulator) 630. The modulated stream of content is then transmittedvia transmission medium 530 to content delivery back-end 500, whichfeatures one or more digital device(s). For instance, the modulatedstream of content may be routed to via a cable plant 635 to contentdelivery back-end 500. The operation performed by content provider 520,including but not limited to the PID mapping for content replacement, iscarried out under control of a control computer 640.

In general, a system can be used to form a content substitution encoderconsistent with certain embodiments of the invention in which input datainclusive of primary content and replacement content. PID mapping logic610 assigns a primary PID to the primary content and assigns a secondaryPID to the replacement content. Private data to identify the primarycontent by the primary PID and replacement content by the secondary PID,along with the primary content mapped to the primary PID and thereplacement content mapped to the secondary PID, are assembled into adata stream.

Of course, it is contemplated that PID mapping logic 610 may beinstructed to remap the PIDs to select only the desired content based onparental control parameters routed to content provider 520 from variousdigital devices. Hence, if utilized, secondary PIDs may be associatedwith previously substituted segments of primary content.

Referring now to FIG. 7, an exemplary diagram of digital device 110 ofthe content delivery system 100 is shown. Digital device 110 comprises apower supply unit 700, a tuner/demodulation unit 710 and data processingunit 720. Power supply unit 700 is adapted to provide powered totuner/demodulation unit 710 and data processing unit 720. Power supplyunit 700 may provide regulated power based on input power received froma wall socket (e.g., 110 volts alternating current “VAC”, 220 VAC,etc.). Tuner/demodulation unit 710 tunes to a channel to receive theprogram and demodulates an incoming program to recover the content.Then, the content is routed as a digital bit stream to data processingunit 720.

According to one embodiment of the invention, data processing unit 720comprises a processor 730 (e.g., central processing unit) and/or adecoder 740. These components may be implemented as separate integratedcircuits or as a single integrated circuit. As described herein, dataprocessing unit 720 features logic operating to decode and processincoming information as well as a content filter unit 750 that isadapted to alternatively select replacement content during playback ofvideo and/or audio.

In general, according to one embodiment of the invention, content filterunit 750 is designed to select alternate PID(s) from the digital bitstream based on findings by data processing unit 720, which accessesmeta data in Adaptation fields within MPEG packets to obtain informationfor use in evaluating whether a scene has objectionable content.

According to one embodiment, data processing unit 720 is adapted toprocess the closed captioning and emergency alert system (EAS) data aswell as guide data. In addition, data processing unit 720 may be adaptedto access the meta data in Adaptation fields of MPEG packets in order toprovide signaling to content filter unit 750 regarding the subjectmatter within each scene that is scheduled to be played back in the nearfuture. This allows content filter unit 750 to determine whether thesubject matter constitutes objectionable content.

Data processing unit 720 (or perhaps a separate graphics component) isadapted for rendering on-screen menus or guide data. For instance, dataprocessing unit 720 may be adapted to collect guide data and for storageof such guide data in local memory (not shown). When such data isrequested, it is displayed and may even overlay video concurrently beingdisplayed. Also, data processing unit 720 may be adapted to distort orblank out displayed images.

Referring to FIGS. 8A and 8B, exemplary diagrams of an Adaptation field810 of an I-frame 800, being a portion of a MPEG packet 805 of thedigital data stream is shown. Adaptation field 810 features controlinformation 815 (also referred to as “private data”) to initiate afiltering operation is shown. According to one embodiment of theinvention, Adaptation field 810 includes control information 815 used bydata processing unit 720 of FIG. 7 to (i) determine if contentreplacement is enabled or disabled and (ii) evaluate whether replacementcontent should be used. Such evaluation is conducted ahead of anyscheduled rendering (playback for visual or audible perception) ofscenes of digital content.

Adaptation field 810 includes control information 815 that features aplurality of sub-fields, including but not limited or restricted to oneor more of the following: (1) PID sub-field 820, (2) Conversion typesub-field 825; (3) Content type sub-field 830; (4) Replacement typesub-field 835; (5) Length sub-field 840; (6) Rating sub-field 845; (7)Tag sub-field 850; (8) Intensity level sub-field 855; (9) Filterblocking summary sub-field 860 and (10) Flag deletion sub-field 865.

As illustrated in FIG. 5B, the PID sub-field 820 is adapted to identifywhat PIDs are provided for a scene started by I-frame 800. These PIDsmay include a single (Primary) PID where no replacement content isavailable, and/or one or more secondary PIDs (e.g., PID-2, PID-3, etc.).

Conversion type sub-field 825 identifies whether the replacement contentassociated with a secondary PID is an up-rating (e.g., increasing therating level and making the program for more mature audiences) or adown-rating (e.g., decreasing the rating level and making the displayedprogram for more general audiences).

Content type sub-field 830 identifies the type of content associatedwith I-frame 800 and the scene associated with I-frame 800. Forinstance, content type sub-field 830 may identify the content as amovie, a television series, a news clip, a live sporting event, a liveentertainment event such as a concert, etc. This information is used toallow content filter unit 750 of FIG. 7 to filter a variety of types ofcontent, especially those types of content that have not been subject toa ratings scheme.

Replacement type sub-field 835 identifies the type of replacement schemeavailable. For instance, the replacement scheme may involve full scenereplacement or inter-scene data replacement (e.g., pixel obfuscation,audio muting, etc.) to reduce transmission bandwidth requirements.Alternatively, the replacement scheme may simply involve the insertionof content markers that are used by personal video recorders (PVRs) orother types of digital devices to adjust content playback duringpost-recording solutions.

Length sub-field 840 identifies the number of frames (I, B, P)associated with the length of the digital data stream forming thecontent.

Rating sub-field 845 provides an age or content rating associated withthe scene that are used to determine if any modification, replacement orpreclusion involving the rendering of a scene is required.

Tag sub-field 850 provided expanded filtering capabilities that enablefiltering to be conducted based on specific types of content found ineach scene and not the overall rating of the scene.

Intensity level sub-field 855 provides an alphanumeric value of theintensity of certain subject matter, such as levels of violence, sex orlanguage for example, to provide greater granularity for the filteringoperation. This granularity may avoid false-positives where some contentmay feature content that is in an objectionable category, but suchinclusion is minimal or not graphic.

Filter blocking summary sub-field 860 provides for a contextual (tag)overrides described below.

Flag deletion sub-field 865 is a flag (or marker) used by PVR andperhaps other recording digital devices to remove blocking content.

III. Content Replacement Techniques

Referring to FIG. 9, an exemplary diagram of a first embodiment of acontent replacement process consistent with certain embodiments of theinvention is shown. For this embodiment, no replacement content isprovided in the digital data stream, but control information is placedwithin the MPEG packets such as within the Adaptation field of theI-frame as described above. In response to detecting objectionablecontent, accomplished by comparing preset parental control parameterswith information contained in rating, tag and/or intensity sub-fields845-855 of FIG. 8B, the content filter unit causes a black frame 900 tobe displayed in lieu of the objectionable content. Alternatively, inlieu of a black screen, a pre-stored banner 910, namely selectedadvertisements paid by various companies having a rating no greater thanthe limits (e.g., ratings, absence of certain subject matter, etc.)established for the parental control parameter(s) (hereinafter referredto as “parental control limits”), may be displayed if the blocking isfor a longer duration.

Referring now to FIG. 10, an exemplary diagram of a second embodiment ofa content replacement process consistent with certain embodiments of theinvention is shown. For this embodiment, primary content 1000 formed bya plurality of primary content segments 1010 ₁-1010 _(N) (where N≧1),which are associated with corresponding primary identifiers (ID-1), suchas a Packet Identifier (PID), and loaded into content filter unit 750 ofFIG. 7. Replacement content 1020, namely replacement content segments1030 ₂, 1030 ₅-1030 ₇ & 1035 ₂ associated with certain secondaryidentifiers (e.g., ID-2, ID-3), is also provided to the content filterunit. Replacement content segments 1030 ₂, 1030 ₅-1030 ₇ & 1035 ₂ may belower rated or higher rated content than corresponding primary contentsegments 1010 ₂ & 1010 ₅-1010 ₇.

In response to detecting objectionable content in primary contentsegments 1010 ₂ and 1010 ₇, replacement content segments 1035 ₂ and 1030₇ are substituted for content segment 1010 ₂ and 1010 ₇, respectively.The collection of primary content segments and replacement contentsegments are output for viewing and/or listening by the end user. Uponevaluation of the primary content segment 1010 ₃, it is determined thatthis also includes objectionable content, but no replacement contentsegment is provided. As a result, screen blanking or audio muting isconducted as described in FIG. 7 as denoted by “X”.

Referring now to FIG. 11, an illustrative flowchart of a process fordetermining whether a segment of content associated with a programshould be prevented from being displayed is shown. As illustrated,initially programmed parental control parameters are examined todetermine what constitutes “objectionable content” for this contentfilter unit (block 1100). The parental control parameters are preset asshown in FIGS. 13, 14 and 15A.

Next, a determination is made whether the content filter unit has beenenabled (block 1110). If not, the V-chip is used as an alternativesource for regulating the content to be viewed (arrow 1115). If thecontent filter unit has been enabled, a first evaluation is whether theprogram being transmitted can be altered and maintained within theparental control limits set for the content filter unit (block 1120).This can be accomplished by analysis of a meta data regarding thecontent that summarizes the ratings (age or content based), contentlabels or content tags on a per scene or per program basis. As a result,replacement content having a lower rating (referred to as “down-rating”)or having a higher rating may be substituted for the primary (chosen)content, provided such content still falls within the parental controllimits (also referred to as “up-rating”).

For each scene, a portion of the incoming content segment, such as theAdaptation field for example, is accessed to retrieve informationtherefrom (block 1130). A determination is made whether the receivedcontent segment includes control information to enable filtering of thecontent (block 1140). If not, the received content segment is renderedwithout filtering (block 1150). If the received content segment includescontrol information, a determination is made whether the primary contentshould be substituted for replacement content (blocks 1160 and 1170).The replacement scheme is set forth in FIG. 12.

FIG. 12 provides an illustrative flowchart of a process for substitutingreplacement content based on parental control parameters. Initially, asshown in block 1200, parental control parameters associated with thescene are examined. Next, a determination is made whether the display ofthe scene is controlled based on an Age Rating (block 1210). If so,Age-based rating processing is conducted where the Age Rating of thescene is compared to Aged-based parental control limits programmed bythe end user and monitored by the content filter unit (block 1215). Inthe event that the Age Rating of the scene exceeds the Aged-basedparental control limits, the scene is not rendered, but instead may bedown-rated through substitution of replacement content or blocked. Thissubstitution may be regulated through use of the synchronization marksdescribed above.

As an option, the content filter unit can be programmed to substitutereplacement content having a higher rating than the primary content. Thehigher rated content would be displayed if the following conditions aremet: (1) replacement content has a higher rating than the primarycontent, and (3) the Age Rating of the replacement content does notexceed the Aged-based parental control limits.

For instance, as shown in FIG. 13, an exemplary ratings table 1300 forobjectionable content is shown. Herein, the content filter unit isprogrammed by setting an Age-based parental control limit within a firstparental control parameter 1310. As an example, the Age Rating isestablished to cover age-based rating levels G, PG, PG-13, R and NC-17ratings used in movies as well as rating levels TV-Y, TV-Y7, TV-Y7-FV,TV-G, TV-14 and TV-MA used in television broadcasts. For instance, TV-14may have the same Age rating level as PG-13 (V,S), where “V” denotesviolence and “S” denotes sexual content.

As an illustrative example, a first scene 1320 of a program may featurePG-13 content as identified by a primary identifier (ID-1), and thus, nosubstitution is conducted if the Aged-based parental control limit isset to “PG-13”. However, if the Aged-based parental control limit is setto allow only “G” (or lesser rated) movies or related televisionbroadcasting (e.g. TV-G), replacement content associated with a firstalternative identifier (ID-2) would be rendered for the first scene.However, if no age threshold was set, the “R” version of the scene wouldbe rendered by substitution of the “PG-13” content associated with ID-1with “R” scene content associated with a second alternative identifier(ID-3). The same operations would occur for each scene.

As a result, for “PG-13” Aged-based parental control limits set withinfirst parental control parameter 1310, content segments for scenes 1-8associated with ID-1 would be rendered. For “G” Aged-based limits setwithin first parental control parameter 1310, content segmentsassociated with ID-1 (scenes 4 and 8) and ID-2 (scenes 1-3 & 5-7) wouldbe shown. Where no Aged-based limits are established and the contentfilter unit is programmed to increase the rating of the content wherepossible, content associated with ID-1 (scene 3), ID-2 (scene 8) andID-3 (scenes 1-2 & 4-7) would be shown.

Referring back to FIG. 12, another determination is whether the displayof the scene is controlled based on content label scheme (blocks 1220and 1225). With respect to content label evaluation scheme, adetermination is made if any particular category of content is present.This is similar to Content Rating scheme describe below where the“Intensity” parental control limits are set to “0”, which also indicatesthat if there is any content with this subject matter, replacementcontent without such subject matter (and within the limits set for othertypes of subject matter) is selected. If no replacement content isavailable, the content is blocked or muted.

Yet another determination is whether the display of the scene iscontrolled based on content rating scheme (block 1230). If so, anIntensity rating is applied to a variety of subject matter categoriesthat parents may find objectionable for their children (block 1235). Forinstance, these categories may be broadly defined such as violence, sex,language, and the like. For those content segments that include thissubject matter, “Intensity” values are assigned for each scene based onthe maturity level of the subject matter.

As an illustrative example, as shown in FIG. 14, if a first scene 1400for primary content 1410 of a program includes a fist fight along with acurse word, the violence rating may be set to a first prescribed value,which will likely be fairly low (e.g., 2 out of 15). Depending on thecurse word spoken and the audible nature of the curse word, the languagerating may be set to a second prescribed value (e.g., 6 out of 15).Replacement content 1412 is provided and associated with ID-2, it mayinclude the elimination of the fist flight and merely a shouting matchwithout any curse words. This replacement content may be substituted ifthe violence parental control parameters for violence and language areless than “2” and “6”, respectively. Otherwise, the primary content isshown since these Intensity values are less than or equal to theparental control limits of “6”, “7” and “3” for violence, language andsex parental control parameters 1420, 1422 and 1424.

Of course, as an alternative, there may be replacement content 1414where the fist fight is substituted with a knife fight as originallyintended by the director for this program. This replacement contentwould have a higher violence rating (e.g., 6 out of 15). Since theparental control limits for violence were set at “6”, replacementcontent associated with ID-3 would be shown if up-rating of the contentis desired.

For these parent control limits, if no up-rating is desired, scenes 1,5-6 and 8 would render content associated with ID-1 (primary ID) andscenes 2-4 and 7 would render content associated with ID-2. With respectto scenes 5 and 7, since no up-rating is desired, the primary content isnormally retained as a default, providing scenes 5 and 7 fall within theparental control limits selected. However, it is contemplated that thedefault setting can be configured to select the content segment having arating closest to the parental control limits.

In the event that up-rating of content has been selected, scenes 2 and 8would render content associated with ID-1 (primary ID), scenes 3-6 wouldrender content associated with ID-2 (first alternative ID) and scenes 1and 7 would render content associated with PID-3 (second alternativeID).

Referring back to FIG. 12, another determination is whether the displayof the scene is controlled based on a tag rating scheme (blocks 1240 and1245). As shown in FIGS. 15A and 15B, the “tag rating” scheme provides asubstantial granularity as to what specific content may be excluded.According to one embodiment, each category 1500-1540 and sub-category isuniquely assigned a unique bit pattern so that a multi-byte tag valueassociated with each scene may be used to identify multiple categorieswhere potential objectionable content may be found. Hence, specificrenderings of violence, language, sex, etc. may be prevented.

As an optional configuration, each category may be assigned a particularbit and sub-categories are identified by bit combinations inclusive ofthe bit assigned to the category. As an example, as shown in FIG. 15B, atag table may be created where a fourth least significant bit(bbb00001xxx) of a double word tag (4-bytes, where “b” denotes byte) isset when the scene involves violence. For the violence categories, anumber of subcategories may be used to denote blood and gore(bbb00001001); sexual assault (bbb00001010), disturbing images(bbb00001011), etc.

Hence, for each scene, the tag is extracted and processed, such asconducting Exclusive OR'ed with the parent control parameter tags set bythe user, in order to determine if any objectionable content is found inthe scene. If not, the original content may be displayed. Otherwise, ifobjectionable content is found, a determination is made whether thereplacement content features the objectionable content. If not, thereplacement content is displayed. Otherwise, if down-rating is needed,no override is indicated, and no replacement content is available, theprimary content is blocked or muted.

Upon determining that the scene includes objectionable content, as anoptional feature, a determination may be conducted to understand thecontext that the objectionable content is featured.

For instance, as an illustrative example, certain nudity may beprevalent for a medical procedure featured in the program. Therefore, ifthe program is a documentary on medical procedures, various bitsreserved in the parental control parameters or separate contextidentifiers coupled be selected as illustrated in FIG. 15B. Hence,certain sub-categories of sex tags may be ignored. Similarly, a gunfight may be prevalent in a documentary regarding the Wild West so anyviolence tags below a predetermined rating (e.g., rating “6”) would beignored. Hence, the context parameter can be used to override thecontent segment replacement operations described above.

In the foregoing description, the invention is described with referenceto specific exemplary embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the present invention asset forth in the appended claims. The specification and drawings areaccordingly to be regarded in an illustrative rather than in arestrictive sense.

1. A nonlinear editing system comprising: a video encoder to receive afirst video stream including video originally intended as video for ascene of a program and a second video stream including a first alternatevideo for replacing the video; an audio compressor to receive a firstaudio stream including audio originally intended as audio for the sceneof the program and a second audio stream including a first alternateaudio for replacing the audio; and an engine unit to synchronize thevideo with the first alternate video and to synchronize the audio withthe first alternate audio so that, where the scene of the programincludes objectionable content, either (i) the video is substituted withthe first alternate video, (ii) the audio is substituted with the firstalternate audio, or (iii) both the video and the audio are substitutedwith the first alternate video and the first alternate audio.
 2. Thenonlinear editing system of claim 1, wherein the video encoder includesa first video encoder to process the first video stream and a secondvideo encoder to process the second video stream.
 3. The nonlinearediting system of claim 1, wherein the second video stream includes ablank screen video between the first alternate video and a secondalternate video.
 4. The nonlinear editing system of claim 3, wherein thesecond audio stream includes muted audio between the first alternateaudio and a second alternate audio.
 5. The nonlinear editing system ofclaim 1, wherein the engine unit uses synchronization marks inserted bythe video encoder and the audio compressor.
 6. The nonlinear editingsystem of claim 4, wherein the engine unit trims the blank screen videofrom the second video stream and the muted audio from the second audiostream.
 7. The nonlinear editing system of claim 6, wherein the engineunit insert a reminder of the second video stream after trimming of theblank screen video to be multiplexed into a primary transport streamincluding the video and the audio.
 8. The nonlinear editing system ofclaim 7, wherein the engine unit insert a reminder of the second audiostream after trimming of the muted audio to be multiplexed into theprimary transport stream.
 9. A nonlinear editing system comprising: avideo encoder to receive a first video stream including video originallyintended as video for a scene of a program and a second video streamincluding a first alternate video for replacing at least a portion ofthe video; an audio compressor to receive a first audio stream includingaudio originally intended as audio for the scene of the program and asecond audio stream including a first alternate audio for replacing atleast a portion of the audio; and an engine unit to determine if thescene of the program includes objectionable content and to substitutemain content, being at least one of the video and the audio, withreplacement content if the scene of the program includes objectionablecontent and the replacement does not feature the objectionable content.10. The nonlinear editing system of claim 9, wherein the replacementcontent includes the alternate video.
 11. The nonlinear editing systemof claim 9, wherein the replacement content includes the alternateaudio.
 12. The nonlinear editing system of claim 9, wherein thereplacement content is one of an ad banner and distorted display images.13. The nonlinear editing system of claim 9, wherein the engine unitdetermining if the scene of the program features objectionable contentby accessing information provided with main content including the videoand the audio and comparing the information with prior establishedparental control limits.
 14. The nonlinear editing system of claim 9,wherein the engine unit accessing information provided with the maincontent and comparing the information with prior established parentalcontrol limits by determining whether an Age-based rating assigned tothe scene of the program associated with the main content exceeds anAge-based parental control limit being the prior established parentalcontrol limit and to substitute the main content with the replacementcontent if (i) the Age-based rating of the scene associated with themain content exceeds the Age-based parental control limit and (ii) anAge-based rating of the replacement content is less than or equal to theAge-based parental control limit.
 15. The nonlinear editing system ofclaim 9, wherein the engine unit comparing of the information byanalyzing tag information provided with the main content for the scene,determining by analysis of the tag information if the main contentincludes content identified as objectionable, and substituting the maincontent with the replacement content if the main content includesobjectionable content as identified by the tag information and thereplacement content does not include the objectionable content.
 16. Amachine readable medium to store software that, if executed by an engineunit, will cause a non-linear editing system to perform the followingoperations: receiving a first video stream including video originallyintended for a scene of a program and a second video stream including afirst alternate video for replacing the video; receiving a first audiostream including audio originally intended for the scene of the programand a second audio stream including a first alternate audio forreplacing the audio; and synchronizing the video with the firstalternate video and to synchronize the audio with the first alternateaudio so that, where the scene of the program includes objectionablecontent, either (i) the video is substituted with the first alternatevideo, (ii) the audio is substituted with the first alternate audio, or(iii) both the video and the audio are substituted with the firstalternate video and the first alternate audio.
 17. The machine readablemedium of claim 16, wherein the synchronizing performed by the softwareincludes trimming blank screen video from the second video stream andmuted audio from the second audio stream.
 18. The machine readablemedium of claim 17, wherein the synchronizing performed by the softwarefurther includes inserting a reminder of the second video stream aftertrimming of the blank screen video to be multiplexed into a primarytransport stream including the video and the audio.
 19. The machinereadable medium of claim 18, wherein the synchronizing performed by thesoftware further includes inserting a reminder of the second audiostream after trimming of the muted audio to be multiplexed into theprimary transport stream.